Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Store everest results in ERT storage #9161

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Store everest results in ERT storage #9161

wants to merge 3 commits into from
+1,163 −31

Conversation

yngve-sk
Copy link
Contributor

@yngve-sk yngve-sk commented Nov 6, 2024 •
edited
Loading

Issue
Resolves #8811

Store it as some dataframes and dump it in storage, not entirely sure how much of it should be saved. Will use everest_data_api as reference for what is required.

Objectives, constraints and control names are pivoted so that each distinct value has its own column, which is more concise and better to serve via an API. It makes each row in the dataset represent the entire set of controls, objectives, and constraints for one batch | batch -> realization | batch -> realization -> perturbation.

objectives_for_batch.to_dicts()
Out[5]: 
[{'result_id': 0,
  'batch_id': 0,
  'total_objective_value': 1886.010009765625,
  'npv_function': 1886.010009765625}]
objectives_per_real.to_dicts()[:4]
Out[6]: 
[{'batch_id': 0,
  'realization': 0,
  'npv_function': 1886.010009765625,
  'group_w14': 0.0623,
  'group_w01': 0.0627,
  'group_w12': 0.0621,
  'group_w09': 0.0618,
  'group_w03': 0.0629,
  'group_w13': 0.0622,
  'group_w07': 0.0633,
  'group_w04': 0.063,
  'group_w02': 0.0628,
  'group_w10': 0.0619,
  'group_w08': 0.0617,
  'group_w05': 0.0631,
  'group_w11': 0.062,
  'group_w15': 0.0624,
  'group_w06': 0.0632,
  'group_w00': 0.0626},
 {'batch_id': 0,
  'realization': 1,
  'npv_function': 1886.010009765625,
  'group_w14': 0.0623,
  'group_w01': 0.0627,
  'group_w12': 0.0621,
  'group_w09': 0.0618,
  'group_w03': 0.0629,
  'group_w13': 0.0622,
  'group_w07': 0.0633,
  'group_w04': 0.063,
  'group_w02': 0.0628,
  'group_w10': 0.0619,
  'group_w08': 0.0617,
  'group_w05': 0.0631,
  'group_w11': 0.062,
  'group_w15': 0.0624,
  'group_w06': 0.0632,
  'group_w00': 0.0626}]

Objective results:

objectives_for_batch
Out[1]: 
shape: (1, 4)
┌───────────┬──────────┬───────────────────────┬──────────────┐
│ result_id ┆ batch_id ┆ total_objective_value ┆ npv_function │
│ ---       ┆ ---      ┆ ---                   ┆ ---          │
│ i64       ┆ i64      ┆ f64                   ┆ f64          │
╞═══════════╪══════════╪═══════════════════════╪══════════════╡
│ 0         ┆ 0        ┆ 1886.01001            ┆ 1886.01001   │
└───────────┴──────────┴───────────────────────┴──────────────┘
objectives_per_real
Out[2]: 
shape: (2, 19)
┌──────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬───────────┐
│ batch_id ┆ realizati ┆ npv_funct ┆ group_w14 ┆ … ┆ group_w11 ┆ group_w15 ┆ group_w06 ┆ group_w00 │
│ ---      ┆ on        ┆ ion       ┆ ---       ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ---       │
│ i64      ┆ ---       ┆ ---       ┆ f64       ┆   ┆ f64       ┆ f64       ┆ f64       ┆ f64       │
│          ┆ i64       ┆ f64       ┆           ┆   ┆           ┆           ┆           ┆           │
╞══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 0        ┆ 0         ┆ 1886.0100 ┆ 0.0623    ┆ … ┆ 0.062     ┆ 0.0624    ┆ 0.0632    ┆ 0.0626    │
│          ┆           ┆ 1         ┆           ┆   ┆           ┆           ┆           ┆           │
│ 0        ┆ 1         ┆ 1886.0100 ┆ 0.0623    ┆ … ┆ 0.062     ┆ 0.0624    ┆ 0.0632    ┆ 0.0626    │
│          ┆           ┆ 1         ┆           ┆   ┆           ┆           ┆           ┆           │
└──────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴───────────┘
constraints_for_batch
Out[3]: 
shape: (1, 66)
┌───────────┬──────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬───────────┐
│ result_id ┆ batch_id ┆ oil_prod_ ┆ oil_prod_ ┆ … ┆ oil_prod_ ┆ oil_prod_ ┆ oil_prod_ ┆ oil_prod_ │
│ ---       ┆ ---      ┆ rate_000  ┆ rate_001  ┆   ┆ rate_012. ┆ rate_013. ┆ rate_014. ┆ rate_015. │
│ i64       ┆ i64      ┆ ---       ┆ ---       ┆   ┆ scaled_vi ┆ scaled_vi ┆ scaled_vi ┆ scaled_vi │
│           ┆          ┆ f64       ┆ f64       ┆   ┆ ola…      ┆ ola…      ┆ ola…      ┆ ola…      │
│           ┆          ┆           ┆           ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ---       │
│           ┆          ┆           ┆           ┆   ┆ f64       ┆ f64       ┆ f64       ┆ f64       │
╞═══════════╪══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 0         ┆ 0        ┆ 1000.0    ┆ 1000.0    ┆ … ┆ 0.133333  ┆ 0.133333  ┆ 0.133333  ┆ 0.133333  │
└───────────┴──────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴───────────┘
constraints_per_real
Out[4]: 
shape: (2, 50)
┌──────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬───────────┐
│ batch_id ┆ realizati ┆ oil_prod_ ┆ oil_prod_ ┆ … ┆ group_w12 ┆ group_w13 ┆ group_w14 ┆ group_w15 │
│ ---      ┆ on        ┆ rate_000  ┆ rate_001  ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ---       │
│ i64      ┆ ---       ┆ ---       ┆ ---       ┆   ┆ f64       ┆ f64       ┆ f64       ┆ f64       │
│          ┆ i64       ┆ f64       ┆ f64       ┆   ┆           ┆           ┆           ┆           │
╞══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 0        ┆ 0         ┆ 6000.0    ┆ 6000.0    ┆ … ┆ 0.0621    ┆ 0.0622    ┆ 0.0623    ┆ 0.0624    │
│ 0        ┆ 1         ┆ 6000.0    ┆ 6000.0    ┆ … ┆ 0.0621    ┆ 0.0622    ┆ 0.0623    ┆ 0.0624    │
└──────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴───────────┘

Gradient results:

obj_gradient_for_batch
Out[4]: 
shape: (1, 4)
┌───────────┬──────────┬──────────────┬────────────────────┐
│ result_id ┆ batch_id ┆ npv_function ┆ npv_function.total │
│ ---       ┆ ---      ┆ ---          ┆ ---                │
│ i64       ┆ i64      ┆ f64          ┆ f64                │
╞═══════════╪══════════╪══════════════╪════════════════════╡
│ 1         ┆ 1        ┆ -0.0         ┆ -0.0               │
└───────────┴──────────┴──────────────┴────────────────────┘
obj_gradient_perturbations
Out[5]: 
shape: (4, 21)
┌───────────┬──────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬───────────┐
│ result_id ┆ batch_id ┆ realizati ┆ perturbat ┆ … ┆ group_w08 ┆ group_w07 ┆ group_w01 ┆ group_w05 │
│ ---       ┆ ---      ┆ on        ┆ ion       ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ---       │
│ i64       ┆ i64      ┆ ---       ┆ ---       ┆   ┆ f64       ┆ f64       ┆ f64       ┆ f64       │
│           ┆          ┆ i64       ┆ i64       ┆   ┆           ┆           ┆           ┆           │
╞═══════════╪══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 1         ┆ 1        ┆ 1         ┆ 0         ┆ … ┆ 0.059048  ┆ 0.068553  ┆ 0.062976  ┆ 0.062975  │
│ 1         ┆ 1        ┆ 0         ┆ 1         ┆ … ┆ 0.061752  ┆ 0.069882  ┆ 0.065536  ┆ 0.061705  │
│ 1         ┆ 1        ┆ 0         ┆ 0         ┆ … ┆ 0.057099  ┆ 0.070677  ┆ 0.055112  ┆ 0.071082  │
│ 1         ┆ 1        ┆ 1         ┆ 1         ┆ … ┆ 0.064173  ┆ 0.066663  ┆ 0.068711  ┆ 0.062269  │
└───────────┴──────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴───────────┘
constraint_gradient_for_batch
Out[6]: 
shape: (1, 33)
┌──────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬───────────┐
│ batch_id ┆ oil_prod_ ┆ oil_prod_ ┆ oil_prod_ ┆ … ┆ oil_prod_ ┆ oil_prod_ ┆ oil_prod_ ┆ oil_prod_ │
│ ---      ┆ rate_004  ┆ rate_003  ┆ rate_006  ┆   ┆ rate_014. ┆ rate_005. ┆ rate_012. ┆ rate_010. │
│ i64      ┆ ---       ┆ ---       ┆ ---       ┆   ┆ scaled    ┆ scaled    ┆ scaled    ┆ scaled    │
│          ┆ f64       ┆ f64       ┆ f64       ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ---       │
│          ┆           ┆           ┆           ┆   ┆ f64       ┆ f64       ┆ f64       ┆ f64       │
╞══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 1        ┆ 0.0       ┆ 0.0       ┆ 0.0       ┆ … ┆ 0.0       ┆ 0.0       ┆ 0.0       ┆ 0.0       │
└──────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴───────────┘
constraint_gradient_perturbations
Out[7]: 
shape: (4, 51)
┌──────────┬───────────┬───────────┬───────────┬───┬───────────┬───────────┬───────────┬───────────┐
│ batch_id ┆ realizati ┆ perturbat ┆ oil_prod_ ┆ … ┆ group_w12 ┆ group_w13 ┆ group_w14 ┆ group_w15 │
│ ---      ┆ on        ┆ ion       ┆ rate_000  ┆   ┆ ---       ┆ ---       ┆ ---       ┆ ---       │
│ i64      ┆ ---       ┆ ---       ┆ ---       ┆   ┆ f64       ┆ f64       ┆ f64       ┆ f64       │
│          ┆ i64       ┆ i64       ┆ f64       ┆   ┆           ┆           ┆           ┆           │
╞══════════╪═══════════╪═══════════╪═══════════╪═══╪═══════════╪═══════════╪═══════════╪═══════════╡
│ 1        ┆ 0         ┆ 0         ┆ 6000.0    ┆ … ┆ 0.066225  ┆ 0.067566  ┆ 0.05967   ┆ 0.062201  │
│ 1        ┆ 0         ┆ 1         ┆ 6000.0    ┆ … ┆ 0.069046  ┆ 0.058626  ┆ 0.061307  ┆ 0.059147  │
│ 1        ┆ 1         ┆ 0         ┆ 6000.0    ┆ … ┆ 0.066712  ┆ 0.066355  ┆ 0.067376  ┆ 0.059584  │
│ 1        ┆ 1         ┆ 1         ┆ 6000.0    ┆ … ┆ 0.065746  ┆ 0.069017  ┆ 0.060187  ┆ 0.06393   │
└──────────┴───────────┴───────────┴───────────┴───┴───────────┴───────────┴───────────┴───────────┘

@codecov-commenter
Copy link

codecov-commenter commented Nov 6, 2024 •
edited
Loading

Codecov Report

Attention: Patch coverage is 40.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 90.72%. Comparing base (721f3c6) to head (f9c476a).

Files with missing lines Patch % Lines
src/ert/storage/local_ensemble.py 33.33% 2 Missing ⚠️
src/ert/run_models/everest_run_model.py 50.00% 1 Missing ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main    #9161      +/-   ##
==========================================
+ Coverage   90.70%   90.72%   +0.02%     
==========================================
  Files         351      351              
  Lines       21903    21908       +5     
==========================================
+ Hits        19867    19876       +9     
+ Misses       2036     2032       -4
Flag Coverage Δ
cli-tests 39.23% <20.00%> (+0.02%) ⬆️
gui-tests 71.73% <20.00%> (-0.04%) ⬇️
performance-tests 49.37% <20.00%> (-0.03%) ⬇️
unit-tests 79.63% <40.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@yngve-sk yngve-sk force-pushed the 24.10.25.store-everest-opt-results-in-ertstorage branch 14 times, most recently from 7081e58 to 32193bd Compare November 13, 2024 11:20
@yngve-sk yngve-sk changed the title (wip) Store everest results in ERT storage Store everest results in ERT storage Nov 13, 2024
@yngve-sk yngve-sk marked this pull request as ready for review November 13, 2024 11:26
@yngve-sk yngve-sk force-pushed the 24.10.25.store-everest-opt-results-in-ertstorage branch 13 times, most recently from 5b54ee7 to dda3db9 Compare November 15, 2024 13:06
@yngve-sk yngve-sk force-pushed the 24.10.25.store-everest-opt-results-in-ertstorage branch 6 times, most recently from fb5948d to a737a84 Compare November 18, 2024 11:21
yngve-sk
yngve-sk commented Nov 18, 2024
View reviewed changes
src/ert/run_models/everest_run_model.py
@@ -437,6 +426,8 @@ def run_experiment(
self._result = OptimalResult.from_seba_optimal_result(
seba_storage.get_optimal_result() # type: ignore
)
optimal_result_from_everstorage = self.ever_storage.get_optimal_result()
assert self._result == optimal_result_from_everstorage
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(This is temporary/sanity check to be removed. If all tests pass with this, it means the storage is producing the same optimal results as the old. )

@yngve-sk yngve-sk force-pushed the 24.10.25.store-everest-opt-results-in-ertstorage branch 3 times, most recently from e68761a to a368add Compare November 19, 2024 15:02
@yngve-sk yngve-sk force-pushed the 24.10.25.store-everest-opt-results-in-ertstorage branch from e7336cd to 3678f8d Compare November 20, 2024 10:40
@yngve-sk yngve-sk force-pushed the 24.10.25.store-everest-opt-results-in-ertstorage branch from 3678f8d to c3efb1d Compare November 20, 2024 10:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Refactor communication and storage of optimization results
2 participants
@yngve-sk @codecov-commenter